News Archive

SDSC-Led Team Sets Records in Simulating Seismic Wave Propagation Across the Earth

Research Team Shatters Two-Second Barrier, Named Finalists for Gordon Bell Prize at SC08

Published 11/24/2008

For Immediate Release

Comment:
Laura Carrington, 858-534-5063 or lcarring@sdsc.edu

Media Contacts:
Jan Zverina, SDSC Communications, 858 534-5111 or jzverina@sdsc.edu
Warren R. Froelich, SDSC Communications, 858 822-3622 or froelich@sdsc.edu

A team led by researchers at the San Diego Supercomputer Center at UC San Diego has successfully completed record-setting, petascale-level simulations of the earth's inner structure, paving the way for seismologists to model seismic wave propagations at frequencies of just over one second - the same frequencies that occur in nature.

Modeling Earth's Enigmatic Core
To learn more about the inner sanctum of the earth's core, seismologists take advantage of one of nature's most destructive forces: earthquakes. Somewhat like the way a CAT scan images the brain, seismologists track seismic wave patterns from earthquakes to model the structure of the earth's core. One of the great challenges is to capture the propagation of high-frequency waves, with periods of 1 to 2 seconds, as they travel across the globe. To simulate this activity, seismologists employ a spectral-element application called SPECFEM3D_GLOBE that uses a fine mesh of hexahedral finite elements, pictured here, and high-performance computers.

Image Credit: D. Komatitsch, Université de Pau; L. Carrington, San Diego Supercomputer Center at UC San Diego.

Results of these latest seismic wave simulations were announced at SC08, the international conference for high-performance computing, where the research by the SDSC-led team was a finalist for Gordon Bell prize, awarded annually for outstanding achievement in high-performance computing applications. The SDSC team includes researchers from the California Institute of Technology; Université de Pau and INRIA, France; and the Institut Universitaire de France, France.

The record runs were completed on the 'Jaguar' system at Oak Ridge National Laboratory (ORNL). The record run broke the two-second barrier by achieving a shortest period of 1.15 seconds and 161 teraflops, using 149,784 cores.

That is the shortest wave period ever obtained in seismic wave propagation, as well as the highest level of parallelism and the first sustained performance of seismic wave propagation exceeding 160 TFlops. (One teraflop equals one trillion calculations per second. To put that in perspective, it would take a person operating a hand-held calculator more than 30,000 years to complete one trillion calculations.)

The latest supercomputer simulations, led by SDSC researcher Laura Carrington, focused on overcoming a key challenge in global seismology: modeling the propagation of seismic waves with frequencies as small as one second. The team broke the previous record set in 2003, which achieved frequencies of 3.5 seconds.

"In breaking the two-second barrier, we were able to model wave propagation clear through the Earth to more accurately predict its structure," said Carrington, who is with SDSC's Performance Modeling and Characterization (PMaC) laboratory. "More significantly, by achieving a frequency just above one second, we were able to duplicate the same frequencies that occur in nature, providing an unprecedented level of resolution that will greatly enhance our understanding of the physics and chemistry of the Earth's interior."

Researchers used a spectral finite-element application called SPECFEM3D_GLOBE for the latest and largest simulations ever done in this area of research. Waves at frequencies of between one and two seconds, generated when major earthquakes with magnitudes measuring 6.5 or more occur, help reveal new information about the 3D structure of the Earth because they are the highest frequency signals that can propagate, or travel, all the way through the Earth, particularly near the core-mantle boundary (CMB), the inner core boundary (ICB), and the enigmatic inner core, which is comprised of solid iron.

There is no need to simulate wave periods of less than one second for seismographic comparisons, because such frequencies signals do not propagate across the entire globe, according to the researchers.

Moreover, the team's research is crucial in helping seismologists better understand the dramatic differences in the complex structure of the Earth's inner core, which appears to be anisotropic, or having unequal physical properties along different axes, with dramatic differences between its eastern and western hemispheres.

Earlier, the research team conducted simulations using a wide range of resources provided by the TeraGrid, the nation's largest open scientific discovery infrastructure linking compute resources among 11 partner sites across the U.S. Runs were conducted using approximately 32,000 cores on the 'Ranger' supercomputer at the Texas Advanced Computing Center (TACC) at The University of Texas in Austin. That run achieved a seismic period length of 1.84 seconds and at a sustained 28.7 TFlops.

Prior to that, the team successfully completed several simulations using TeraGrid supercomputer resources, including 29,000 cores on the 'Jaguar' system at ORNL's National Center for Computational Sciences, achieving 1.94 seconds at 35.7 TFlops. Runs were also conducted on the Cray XT4 'Franklin' system at the National Energy Research Scientific Computing Center (NERSC) at the Lawrence Berkeley National Laboratory in Berkeley, Calif.; (12,150 cores with a shortest seismic wave of 3 seconds at 24 TFlops), and on the Cray XT4 'Kraken' system at the University of Tennessee-Knoxville (17,496 cores at 2.52 seconds at 22.4T Flops. The TFlops number in these and subsequent runs were measures using PSiNSlight, a performance measurement and tracing tool developed by SDSC's PMaC Lab.

The researchers also made a number of radical structure and memory management changes to
the SPECFEM3D_GLOBAL application to enable what researchers call "peta-scale-ability", or using processors numbering in the hundreds of thousands. Efforts focused on the application to overlapping communications using non blocking MPI calls, optimizations to reduce cache misses, optimization to reduce I/O, large restructurings to reducing memory consumption, and work to reduce memory fragmentation.

In addition to Carrington, SDSC researchers on the team include Allan Snavely, director of the PMaC Lab; and researchers Michael Laurenzano and Mustafa Tikir. Additional team members include Dimitri Komatitsch, David Michéa, and Nicolas Le Goff of the Université de Pau, CNRS and INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences, Pau, France; and Jeroen Tromp, formerly a professor of geophysics with the California Institute of Technology's Seismological Laboratory. Tromp recently joined Princeton University's geosciences and mathematics departments. Komatitsch is also affiliated with the Institut Universitaire de France, in Paris.

Komatitsch and Tromp developed SPECFEM3D_GLOBE over 10 years ago, in order to do 3D whole Earth models that account for geological variations in and under the Earth's crust. These variations have a dramatic effect on how earthquakes propagate. Komatitsch's work on the application was vital in the collaboration that accomplished this record breaking run.

About the run, Tromp and Komatitsch had this to say: "One of the long-term goals of seismology is to be able to routinely simulate 3D seismic wave propagation in the whole Earth following earthquakes at frequencies of about 1 Hz, the highest frequency signal that can be seen clear across the planet. Very large numerical simulations performed on the new Cray XT5 system at Oak Ridge will enable us to get increasingly closer to this lofty goal."

Research for the SPECFEM3D_GLOBE applications was supported in part by the National Science Foundation (NSF); the Performance Engineering Research Institute (PERI); and the U.S. Department of Energy (DoE); in addition to resources from the NSF-funded TeraGrid program.

The SC08 conference was held in Austin, Texas, November 15-21.

About SDSC
Since being founded in 1985 through a grant from the National Science Foundation as one of the original academic, national-level supercomputer centers, the San Diego Supercomputer Center at UC San Diego has earned a reputation as a leading resource for data-intensive research and expertise, involved in a broad range of studies ranging from the inner working of the human body to the outer reaches of space. In the past year alone, SDSC was home to almost 120 funded projects, while participating as an integral partner in another 34 projects based elsewhere. The Center continues to work closely with the NSF on a wide scope of research initiatives, as well as collaborating with UC San Diego and the entire UC system on numerous projects.

News Archive

SDSC-Led Team Sets Records in Simulating Seismic Wave Propagation Across the Earth

Research Team Shatters Two-Second Barrier, Named Finalists for Gordon Bell Prize at SC08

Related Links